Skip to content

feat: tiered data storage#7

Open
enigbe wants to merge 7 commits into
mainfrom
2025-10-tiered-data-storage
Open

feat: tiered data storage#7
enigbe wants to merge 7 commits into
mainfrom
2025-10-tiered-data-storage

Conversation

@enigbe

@enigbe enigbe commented Oct 21, 2025

Copy link
Copy Markdown
Owner

What this PR does:

We introduce TierStore, a KVStore implementation that manages data across
three distinct storage layers.

The layers are:

  1. Primary: The main/remote data store.
  2. Ephemeral: A secondary store for non-critical, easily-rebuildable data
    (e.g., network graph). This tier aims to improve latency by leveraging a
    local KVStore designed for fast/local access.
  3. Backup: A tertiary store for disaster recovery. Backup operations are sent
    asynchronously/lazily to avoid blocking primary store operations.

We also permit the configuration of Node with these stores allowing
callers to set exponential back-off parameters, as well as backup and ephemeral
stores, and to build the Node with TierStore's primary store. These configuration
options also extend to our foreign interface, allowing bindings target to build the
Node with their own ffi::KVStore implementations.

A sample Python implementation is added and tested.

Additionally, we add comprehensive testing for TierStore by introducing

  1. Unit tests for TierStore core functionality.
  2. Integration tests for Node built with tiered storage.
  3. Python FFI tests for foreign ffi::KVStore implementations.

Concerns

It is worth considering the way retry logic is handled, especially because of nested
retries. TierStore comes with a basic one by default but there are KVStore implementations
that come with them baked-in (e.g. VssStore), and thus would have no need for
the wrapper-store's own logic.

@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 9 times, most recently from 29f47f3 to 264aa7f Compare November 4, 2025 22:07
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 3 times, most recently from a30cbfb to 1e7bdbc Compare December 4, 2025 23:30
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 5 times, most recently from b5e980f to 67d47c2 Compare February 4, 2026 16:28
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 2 times, most recently from 95285b0 to 4b2d345 Compare February 18, 2026 11:23
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 2 times, most recently from cba29a3 to db1fe83 Compare February 24, 2026 23:03
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch from db1fe83 to 35f9ec2 Compare March 9, 2026 15:47
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 3 times, most recently from e89ada5 to 3abd0a5 Compare April 1, 2026 21:44
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 5 times, most recently from 8f122cb to 7155c40 Compare April 6, 2026 10:15
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 4 times, most recently from 7720935 to 181db03 Compare April 9, 2026 07:24
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 5 times, most recently from 5da76a8 to dec4aa7 Compare April 27, 2026 08:52
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 3 times, most recently from c1563e3 to a2458e4 Compare May 6, 2026 07:37
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch from 8dbb312 to 0ae61b7 Compare May 19, 2026 06:05
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch 7 times, most recently from cadf81f to 1f2cd4d Compare June 16, 2026 08:34
enigbe added 3 commits June 19, 2026 23:26
This commit adds `TierStore`, a tiered `KVStore` implementation that
routes node persistence across three storage roles:

- a primary store for durable, authoritative data
- an optional backup store for a second durable copy of primary-backed data
- an optional ephemeral store for rebuildable cached data such as the
  network graph and scorer

TierStore routes ephemeral cache data to the ephemeral store when
configured, while durable data remains primary+backup. Reads and lists
do not consult the backup store during normal operation.

For primary+backup writes and removals, this implementation treats the
backup store as part of the persistence success path rather than as a
best-effort background mirror. Earlier designs used asynchronous backup
queueing to avoid blocking the primary path, but that weakens the
durability contract by allowing primary success to be reported before
backup persistence has completed. TierStore now issues primary and backup
operations together and only returns success once both complete.

This gives callers a clearer persistence guarantee when a backup store is
configured: acknowledged primary+backup mutations have been attempted
against both durable stores. The tradeoff is that dual-store operations
are not atomic across stores, so an error may still be returned after one
store has already been updated.

Additionally, adds unit coverage for the current contract, including:
- basic read/write/remove/list persistence
- routing of ephemeral data away from the primary store
- backup participation in the foreground success path for writes and removals
Add native builder support for tiered storage by introducing
`TierStoreConfig` and builder methods for configuring ephemeral storage
and a local SQLite backup mirror.

During node construction, wrap the configured primary store in
`TierStore` and attach secondary tiers for cache-like ephemeral data and
mirrored durable backup writes. The builder constructs the backup  and
ephemeral stores internally using a dedicated SQLite database file.

Add test coverage for full-cycle backup mirroring, same-path rejection,
and UniFFI-backed builder configuration. Update `setup_builder!` so
FFI-backed builder tests can use mutable configuration helpers.
…ightningdevkit#871

- Added temporary #[allow(dead_code)] annotations to NodeBuilder::set_backup_storage_dir_path
  and NodeBuilder::set_ephemeral_storage_dir_path
- Keeps these tiered storage builder setters compilable while they remain unused before dropping
  in lightningdevkit#871
@enigbe enigbe force-pushed the 2025-10-tiered-data-storage branch from 1f2cd4d to d424891 Compare June 19, 2026 23:38
enigbe added 4 commits June 20, 2026 01:38
avoid buffer clone if backup isn't configured
Here we avoid error-level logging for primary read
misses and log partial primary/backup mutation failures
as possible divergence.
Preserve call-time ordering for TierStore writes and removals by assigning
per-key versions before returning the future.

Also avoid error-level logging for primary read misses, and log partial
primary/backup mutation failures as possible divergence.
Skip the Rust-only backup-store builder test when UniFFI
bindings are enabled, since backup-store configuration
is not exposed through FFI yet (pending lightningdevkit#871).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant